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1.  INTRODUCTION 


The  potential  use  of  automated  speech  recognition  technology  as  a  natural,  alternative 
method  for  the  management  of  aircraft  subsystems  has  been  studied  by  the  Air  Force 
for  over  10  years,  but  because  recognition  accuracies  have  not  attained  acceptable 
levels  for  use  in  the  cockpit,  this  technology  has  not  yet  become  operational  in  that 
environment.  Recent  \work  in  the  area  of  "parallel  recognition"  in  addition  to  natural 
advances  in  computational  speed  and  signal  processing  techniques  have  resulted  in 
significant  increases  in  recognition  accuracies,  spawning  a  renewed  interest  in  the 
application  of  this  technology  to  the  pilot-vehicle  interface  paradigm. 

Speech  recognition  has  long  been  advocated  as  a  natural  and  intuitive  method  by 
which  humans  could  potentially  communicate  with  complex  systems.  One  such  system 
is  the  modern  fighter  aircraft.  As  more  sophisticated  avionics  are  incorporated,  the 
challenge  of  managing  all  the  various  information  sources  becomes  very  critical.  From 
on  and  off-board  sensor  manipulation  to  weapons,  communications  and  navigation 
control,  the  single  seat  fighter  pilot  has  limited  ability  to  effectively  manage  all  of  the 
various  information  available  using  just  hands  and  eyes.  For  these  reasons, 
researchers  have  been  exploring  the  possibilities  of  using  speech  recognition 
technology  to  augment  the  pilot’s  ability  to  control  and  display  information  in  the  cockpit 
(Lizza  and  Goulet,  1986;  Williamson  and  Barry,  1990). 

The  majority  of  speech  recognition  research  in  the  laboratory  and  flight  test 
environments  has  centered  around  the  use  of  discrete  word  recognition  systems, 
which  require  short  pauses  between  each  word  in  the  phrase.  Natural  advances  in 
computational  speed  and  signal  processing  techniques  have  resulted  in  significant 
increases  in  the  word  recognition  accuracies  of  these  devices,  which  recognize  the 
single  words  or  utterances.  Additional  work  in  the  area  of  "parallel  recognition"  has 
further  demonstrated  the  promise  of  this  discrete  word  recognition  technology  to 
facilitate  simple  communication  between  the  pilot  and  the  computers  which  control  the 
myriad  of  subsystems  aboard  the  aircraft  (Barry,  Liggett,  Williamson  and  Raising,  1992; 
Barry,  Solz,  Raising  and  Williamson,  1994). 

More  recent  work  has  focused  on  newer-generation  connected  speech  recognition 
systems,  which  do  not  require  artificial  pauses  to  be  placed  between  the  words  in  a 
phrase.  These  systems  show  tremendous  potential  for  providing  the  pilot  with  the 
ability  to  use  longer  and  more  complicated  sentences  when  communicating  with  the 
aircraft,  while  yielding  phrase  accuracies  above  95%.  Current  research  efforts  are 
focused  on  allowing  the  pilot  more  word  and  phrase  flexibility  when  using  this 
connected  speech  recognition  technology.  With  the  multiple  options  all  representing 
one  single  pilot  intention  however,  intent  accuracy  becomes  meaningful.  The  eventual 
payoff  of  this  research  will  be  the  attainment  of  the  ultimate  pilot-vehicle  interface 
research  goal:  a  natural-language  interface  between  the  pilot  and  his/her  aircraft. 
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This  report  describes  the  results  of  a  laboratory  test  conducted  to  determine  the  word 
and  phrase  accuracy  of  a  connected  speech  recognition  device  and  to  investigate  the 
use  of  alternate  syntax  phrases  to  yield  increased  intent  accuracy. 
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2.  METHOD 


2.1  Objectives 

The  objective  of  this  study  was  to  determine  the  connected  speech  recognition 
accuracy  of  the  ITT-1290  Speech  Recognition  System  using  a  vocabulary  structure 
containing  relatively  "simple"  and  "complex"  phrases. 


2.2  Subjects 

Eleven  subjects  from  the  Cockpit  Integration  Division  took  part  in  the  study.  Since 
previous  research  in  the  speech  recognition  area  revealed  no  significant  gender 
differences  in  recognition  accuracy,  the  subject  pool  consisted  of  both  male  and  female 
participants.  No  tests  for  gender  differences  were  planned. 


2.3  Materials 

The  vocabulary  used  to  test  the  recognition  accuracy  of  the  ITT-1290  system  consisted 
of  49  words  that  are  likely  to  be  used  in  the  cockpit  environment.  The  49  word 
vocabulary  can  be  found  in  Appendix  A.  These  words  were  combined  to  create  a  total 
of  392  stimulus  phrases  that  were  presented  to  the  subjects  for  recognition.  The 
manner  in  which  these  words  were  combined  defined  the  vocabulary  phrase  structure. 
Appendix  B.  presents  the  vocabulary  phrase  structure  for  this  study. 

Two  different  factors  were  considered  in  the  creation  of  the  test  phrases.  These  two 
factors  were  COMPLEXITY  and  NUMBER  OF  ALTERNATES. 


2.3.1  Phrase  Complexity 

Phrase  Complexity  defined  the  number  of  individual  vocabulary  words  that  made  up  the 
phrase.  Of  the  392  stimulus  phrases,  half  (196)  of  them  were  SIMPLE  phrases  and  the 
other  half  were  COMPLEX  phrases.  For  this  study.  Simple  phrases  were  defined  as 
those  phrases  that  contained  less  than  5  vocabulary  words,  while  Complex  phrases 
contained  5  or  more  vocabulary  words.  For  example,  the  phrase 

"change  waypoint  eight" 

was  a  Simple  phrase  because  it  contained  less  than  5  vocabulary  words.  The  phrase 

"north  three  four  seven  six  point  one  two" 
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was  a  Complex  phrase  because  it  contained  5  or  more  vocabulary  words. 


2.3.2  Number  of  Alternates 

In  an  aircraft  cockpit  application,  the  recognition  of  a  spoken  phrase  would  result  in 
some  action  being  taken  by  the  aircraft.  For  example,  recognition  of  the  phrase 

"configure  for  landing" 

would  result  in  the  computer  sending  the  appropriate  command  to  the  aircraft  to  change 
its  configuration  in  preparation  for  landing.  There  were  other  phrases  in  the  phrase  set 
that  also  would  result  in  the  identical  command  being  sent  to  the  aircraft  when 
recognized.  The  phrase 

"setup  landing" 

also  would  result  in  the  aircraft  changing  its  configuration  in  preparation  for  landing. 
These  two  phrases  were  said  to  be  ALTERNATE  phrases.  Of  the  392  phrases  in  the 
study,  147  phrases  were  Alternate  phrases  -  those  having  at  least  one  other  alternate  in 
the  phrase  set.  The  remaining  245  phrases  had  no  alternates  (i.e.  no  other  phrase  in 
the  set  could  produce  an  identical  aircraft  action.) 

Table  1  summarizes  the  make-up  of  the  392  stimulus  phrases  that  were  presented  to 
the  subjects  in  the  this  study.  Of  the  196  Simple  phrases,  147  had  Alternates,  while 
the  remaining  49  had  no  Alternates.  None  of  the  196  Complex  phrases  had 
Alternates. 


Table!  Distribution  of  Phrase  Types  Used  in  Each  Trial 


Phrase  Complexity 

Simple 

Complex 

Total 

Phrases 

Number  of 
Alternates 

At  least  One 
Alternate 

147 

0 

147 

No  Alternates 

49 

245 

Total  Phrases 

196 

392 

Appendix  C  contains  a  complete  list  of  the  392  phrases  used  in  this  study. 

2.4  Hardware 

The  ITT-1290  speech  recognition  system  was  hosted  in  a  Gateway  2000  4DX2-50V 
computer  system.  A  Plantronics  SNC1 657-01  microphone  was  used  for  speech  input. 
The  host  computer  monitor  was  used  to  present  stimulus  phrases  to  the  subject  for 
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The  ITT-1290  speech  recognition  system  was  hosted  in  a  Gateway  2000  4DX2-50V 
computer  system.  A  Plantronics  SNC1 657-01  microphone  was  used  for  speech  input. 
The  host  computer  monitor  was  used  to  present  stimulus  phrases  to  the  subject  for 
recognition.  Recognition  results  were  presented  on  the  computer  monitor  and  also 
stored  in  a  data  file  on  disk. 


2.5  Software 


The  ITT  TGS  program  supplied  with  the  speech  recognition  hardware  was  used  to  train 
the  subjects.  The  following  custom  software  programs  written  specifically  for  this 
experiment  were  also  used; 


Program  Name  Program  Description 


COMPLEXP.EXE 
ITT_CONN.EXE 
RANDOM.EXE 
RANDSORT.EXE 
SCORE  CO.EXE 


Randomly  generated  complex  phrases  based  on 
latitude  /  longitude  digit  strings 
Presented  phrases  to  be  spoken  and  stored  results 
on  disk. 

Generated  random  seeds  used  by  the  RANDSORT 
program. 

Read  phrases  from  a  file  and  wrote  them  back  to 
disk  in  random  order. 

Read  the  raw  data  produced  by  the  ITT_CONN 
program,  scored  the  data,  and  created  a  summary 
data  file  used  by  SPSS  for  data  analysis. 


2.6  Experimental  Design 

The  experimental  design  was  a  3  x  3  x  2  complete  factorial  Within  Subjects  design  with 
three  levels  of  Phrase  Type  (Simple  Alternate,  Simple  No-Alternate,  Complex  No- 
Alternate),  three  Presentation  Sets,  and  two  separate  Trials.  Program  RANDSORT 
used  the  seeds  in  Table  2  to  randomly  re-order  the  392  phrases  presented  to  each 
subject  per  Trial. 

After  the  phrases  were  re-ordered,  they  were  grouped  into  3  sets  (Set  1  contained  131 
phrases.  Set  2  contained  131,  and  Set  3  contained  130).  In  Trial  1,  the  subjects  were 
presented  with  the  phrases  in  Set  1,  then  Set  2,  then  Set  3,  while  in  Trial  2,  the  phrases 
in  Set  3  were  presented  first,  followed  by  the  phrases  in  Set  2  and  then  Set  1 .  The  two 
data  collection  trials  were  run  approximately  10  days  apart.  The  data  matrix  is  shown  in 
Table  3. 
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Table  2.  Seeds  Used  to  Randomize  Phrases 


Subject  Number 

Random  Seed  used  by  RANDSORT  program 

1 

11357 

2 

12014 

3 

20865 

4 

22513 

5 

22940 

6 

17563 

7 

1171 

8 

10731 

9 

26229 

10 

57384 

11 

56833 

Table  3.  Data  Matrix 


2.7  Dependent  Measures 

Three  speech  recognition  accuracy  values  were  computed  for  this  study.  They  were: 


6 


Word  Accuracy  =  Total  words  correct  /  Total 

words  presented 

Phrase  Accuracy  =  Total  phrases  correct  /  Total 

phrases  presented 

Intent  Accuracy  =  Total  resulting  actions 

correct  /  Total  actions 
presented 

To  further  explain  the  meaning  of  these  three  dependent  measures,  consider  the 
following  example: 

Phrase  presented:  Configure  for  Takeoff 

Phrase  recognized:  Setup  Takeoff 

A  total  of  3  words  were  presented,  with  only  the  word  "Takeoff'  being  correctly 
recognized.  Word  accuracy  for  this  phrase  was  therefore  1  /  3  =  33  percent. 

Word  Accuracy  =  1  /  3  =  33  percent 

One  phrase  was  presented,  but  the  wrong  phrase  was  recognized.  Phrase  accuracy  is 
therefore  0/1=0  percent. 

Phrase  Accuracy  =  0/3  =  0  percent 

Even  though  the  wrong  phrase  was  recognized,  the  resulting  action  in  the  cockpit  would 
still  have  been  the  same  as  if  the  correct  phrase  had  been  recognized,  that  being  a 
reconfiguration  of  the  aircraft  to  prepare  for  takeoff.  Intent  accuracy  therefore  is  1  / 1  = 
100  percent. 

Intent  Accuracy  =  1/1  =  100  percent 

The  use  of  intent  accuracy  is  actually  only  meaningful  for  Alternate  phrases  which,  as 
described  in  a  previous  section,  were  those  phrases  that  provided  an  alternative 
method  for  accomplishing  the  same  action.  The  phrases  "configure  for  takeoff'  and 
"setup  takeoff'  both  resulted  in  the  same  action  and  were  therefore  candidates  for 
Intent  Accuracy  analysis.  In  the  case  of  the  phrase  "select  Maverick",  there  was  no 
other  phrase  in  the  entire  syntax  that  could  result  in  a  selection  of  the  Maverick  missile. 
In  these  cases.  Intent  Accuracy  and  Phrase  Accuracy  were  identical.  Of  the  392 
phrases  presented  in  each  trial,  only  the  147  Alternate  phrases  were  used  in  the 
computation  of  Intent  Accuracy. 


2.8  Data  Analyses 

The  3x3x2  factorial  Within  Subjects  experimental  design  is  represented  in  Figure  1. 
The  numbers  in  parentheses  are  the  Degrees  of  Freedom  (DF)  for  each  effect.  From 
the  proposed  experimental  design,  the  Preliminary  ANOVA  Summary  Table  can  be 
derived  and  is  presented  in  Table  4. 
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SS  Total 
(197) 


rSS  Subjects  (S’s) 
(10) 


SS  Repeated 
Measures 
(17) 


» -  SSRMxS’s 

(170) 


-  SS  Trial  (1) 

-  SSSet(2) 

-  SS  Phrase  Type  (2) 

■  —  SS  Trial  x  Set  (2) 

-  SS  Trial  x  Phrase  Type  (2) 

-  SS  Set  X  Phrase  Type  (4) 

-  SS  Trial  x  Set  x  Phrase  type  (4) 

- SS  Trial  xS’s  (10) 

- SS  SetxS's  (20) 

- SS  Phrase  Type  x  S's  (20) 

- SS  Trial  X  Set  x  S's  (20) 

- SS  Trial  X  Phrase  Type  x  S's  (20) 

- SS  Set  X  Phrase  Type  x  S's  (40) 

- SS  Trial  x  Set  x  Phrase  Type  x  S's  (40) 


Figure  1 .  3x3x2  Factorial  Within  Subjects  Experimental  Design 


Table  4.  Preliminary  ANOVA  Summary  Table 


Source 

DE 

Error  Term 

DFE 

Fcrit 

S 

I 

im 

Trial 

1 

Trial  x  S's 

10 

4.96 

Set 

2 

Set  X  S's 

20 

3.49 

Phrase  Type 

2 

Phrase  Type  x  S's 

20 

3.49 

Trial  x  Set 

2 

Trial  x  Set  x  S's 

20 

3.49 

Trial  x  Phrase  Type 

2 

Trial  x  Phrase  Type  x  S's 

20 

3.49 

Set  X  Phrase  Type 

4 

Set  X  Phrase  Type  x  S's 

40 

2.61 

Trial  x  Set  x  Phrase  Type 

4 

Trial  x  Set  x  Phrase  Type  x  S's 

40 

2.61 

2.9  Procedures 

The  first  experimental  session  (Trial  1)  for  each  subject  was  scheduled  to  last  for 
approximately  60  minutes.  The  tasks  accomplished  during  this  period  are  shown  in 
Table  5. 

For  the  first  5  minutes,  subjects  were  briefed  on  the  nature  of  the  experiment. 
Appendix  D  presents  the  subject  briefing.  Subjects  were  shown  the  experimental  setup 
and  were  given  a  brief  explanation  of  how  speech  recognition  works.  Any  questions 
were  answered.  Following  the  initial  briefing,  the  subjects  "trained"  the  speech  system 
to  collect  the  speech  templates  for  use  in  the  recognition  portion  of  the  experiment. 
During  training,  the  subjects  received  some  practice  in  recognition  while  the 
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experimenter  tested  the  validity  of  the  templates.  The  subjects  were  given  a  short 
break  following  the  template  training  session. 

Table  5.  Experiment  Time  Line 
Minutes  from  start 

0  10  20  30  40  50  60 


Briefing  | — | 

Train  #1  1 

Break 

Set#1 

Set  #2 

Set  #3 


During  the  data  collection  session,  the  phrases  contained  in  the  first  set  were 
presented,  one  at  a  time,  on  the  computer  screen  in  front  of  the  subject.  The  subjects’ 
task  was  to  simply  say  the  phrase.  The  ITT-1290  speech  recognition  system  received 
the  speech  signal  and  attempted  recognition  of  each  word  in  the  phrase.  If  the  subject 
made  a  mistake  in  reading  the  prompted  phrase,  the  phrase  was  simply  presented 
again.  When  all  phrases  within  a  set  were  presented,  the  subject  was  given  a  1-  or  2- 
minute  break  while  the  experimenter  reconfigured  the  system  in  preparation  for  the 
presentation  of  the  next  set  of  phrases. 

Data  collection  for  Trial  2  occurred  approximately  10  days  after  the  subject  completed 
Trial  1.  The  data  collection  procedures  used  in  Trial  1  were  again  used  in  Trial  2,  with 
the  exception  that  the  speech  recognition  system  was  not  retrained.  The  recognition 
templates  generated  in  preparation  for  Trial  1  were  used  during  recognition  in  Trial  2. 
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3.  RESULTS 


Recognition  data  for  both  Trials  were  collected  for  all  11  subjects  over  a  three  week 
period  of  time.  Program  SCORE_CO,  written  specifically  for  this  study,  was  used  to 
score  the  raw  data  and  to  generate  summary  statistics  that  could  be  analyzed  using  the 
SPSS  software.  Appendix  E  contains  the  summary  data  generated  by  the  program  for 
each  subject,  along  with  a  one-page  table  summarizing  the  overall  results  of  the  test. 

SPSS  program  analyses  were  done  to  test  the  experimental  effects  of  Trial,  Set  order, 
and  Phrase  Type  on  the  3  dependent  measures  of  Word  Accuracy,  Phrase  Accuracy, 
and  Intent  Accuracy. 


3.1  Word  Accuracy 

The  results  of  the  Analysis  of  Variance  for  Word  Accuracy,  summarized  in  Table  6, 
revealed  a  significant  main  effect  for  Phrase  Type  (F  =  19.03,  p  <  0.000).  That  is, 
different  word  accuracies  were  obtained  by  the  Alternate,  No  Alternate  and  Complex 
phrases.  Orthogonal,  pairwise  comparisons  of  the  Phrase  Type  revealed  significantly 
better  word  recognition  accuracies  for  Complex  phrases  than  for  Alternate  phrases  (F  = 
21.78,  p  <  0.001)  and  better  accuracies  for  No  Alternate  phrases  compared  with 
Alternate  phrases  (F  =  24.65,  p  <  0.001),  as  shown  in  Figure  2.  No  other  significant 
effects  were  observed. 


Table  6.  ANOVA  Summary  Table  for  Word  Accuracy 


Source 

ss 

DE 

MS 

E 

Sig 

of  F 

Trial 

0.24 

1 

0.24 

0.22 

P< 

0.649 

Set 

0.87 

2 

0.43 

0.54 

P< 

0.588 

Phrase  Type 

167.55 

2 

83.77 

19.03 

P< 

0.000*** 

Trial  X  Set 

1.63 

2 

0.82 

1.09 

P< 

0.355 

Trial  x  Phrase  Type 

4.06 

2 

2.03 

1.73 

P< 

0.203 

Set  X  Phrase  Type 

0.61 

4 

0.15 

0.20 

P< 

0.936 

Trial  x  Set  x  Phrase  Type 

5.10 

4 

1.28 

1.35 

P< 

0.267 

Complex  vs  No  Alternate 

0.12 

1 

0.12 

0.24 

P< 

0.632 

comparison 

Complex  vs  Alternate  comparison 

18.97 

1 

18.97 

21.78 

P< 

0.001*** 

No  Alternate  vs  Alternate 

22.10 

1 

22.10 

24.65 

P< 

0.001*** 

comparison 
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Word  Accuracy 


Complex  No  Alternate  Alternate 


Phrase  Type 

Figure  2.  Word  Accuracy  Comparisons  for  Phrase  Type 


3.2  Phrase  Accuracy 

The  results  of  the  Analysis  of  Variance  for  Phrase  Accuracy  are  summarized  in  Table  7. 
A  significant  main  effect  for  Phrase  Type  (F  =  15.46,  p  <  0.000)  was  observed, 
indicating  that  different  phrase  accuracies  were  obtained  by  the  Alternate,  No  Alternate 
and  Complex  phrases.  Orthogonal,  pairwise  comparisons  of  the  Phrase  Type  revealed 
significantly  better  phrase  recognition  accuracies  for  No  Alternate  phrases  than  for 
Complex  phrases  (F  =  11.12,  p  <  0.008)  and  better  accuracies  for  No  Alternate  phrases 
compared  with  Alternate  phrases  (F  =  34.18,  p  <  0.000),  as  shown  in  Figure  3.  No 
other  significant  effects  were  observed. 


3.3  Intent  Accuracy 

Table  8  shows  the  results  of  the  Analysis  of  Variance  for  Intent  Accuracy.  A  significant 
main  effect  for  Phrase  Type  (F  =  9.75,  p  <  0.001)  was  observed,  indicating  that  different 
intent  accuracies  were  obtained  by  the  Alternate,  No  Alternate  and  Complex  phrases. 
Orthogonal,  pairwise  comparisons  of  the  Phrase  Type  revealed  significantly  better 
intent  recognition  accuracies  for  No  Alternate  phrases  than  for  Complex  phrases  (F  = 
11.12,  p  <  0.008)  and  better  accuracies  for  Alternate  phrases  compared  with  Complex 
phrasU  (F  =  12.13,  p  <  0.006),  as  shown  in  Figure  4.  No  other  significant  effects  were 
observed. 
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Table  7.  ANOVA  Summary  Table  for  Phrase  Accuracy 


Source 

ss 

DF 

MS 

£ 

Sig  of  F 

Trial 

0.03 

1 

0.03 

0.00 

p  <  0.964 

Set 

10.71 

2 

5.35 

0.70 

p  <  0.509 

Phrase  Type 

916.82 

2 

458.41 

15.46 

p  <  0.000*** 

Trial  x  Set 

6.68 

2 

3.34 

0.62 

P  <  0.547 

Trial  x  Phrase  Type 

23.59 

2 

11.79 

1.44 

p  <  0.260 

Set  X  Phrase  Type 

9.99 

4 

2.50 

0.37 

p  <  0.831 

Trial  x  Set  x  Phrase  Type 

38.60 

4 

9.65 

0.75 

p  <  0.567 

Complex  vs  No  Alternate 

63.92 

1 

63.92 

11.12 

p  <  0.008*** 

comparison 

Complex  vs  Alternate  comparison 

16.34 

1 

16.34 

3.16 

p<  0.106 

No  Alternate  vs  Alternate 

114.91 

1 

114.91 

34.18 

p  <  0.000*** 

comparison 


Phrase  Accuracy 


100 
C?  99 
^  98 
2  97 

i  96 
1  95 

«  94 

pL,  93 
92 


98.79% 


Complex  No  Alternate  Alternate 

Phrase  Type 

Figure  3.  Phrase  Accuracy  Comparisons  for  Phrase  Type 
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Table  8.  ANOVA  Summary  Table  for  Intent  Accuracy 


Source 

SS 

DF 

MS 

E 

Sig  of  F 

Trial 

0.64 

1 

0.64 

0.07 

p  <  0.791 

Set 

9.16 

2 

4.58 

0.88 

p  <  0.430 

Phrase  Type 

545.83 

2 

272.92 

9.75 

p  <  O.OOT 

Trial  x  Set 

7.16 

2 

3.58 

0.63 

p  <  0.541 

Trial  x  Phrase  Type 

18.35 

2 

9.17 

1.24 

p<  0.311 

Set  X  Phrase  Type 

8.67 

4 

2.17 

0.43 

p  <  0.788 

Trial  x  Set  x  Phrase  Type 

23.73 

4 

5.93 

0.79 

p  <  0.537 

Complex  vs  No  Alternate 

63.92 

1 

63.92 

11.12 

p  <  0.008 

comparison 

Complex  vs  Alternate  comparison 

72.30 

1 

72.30 

12.13 

p  <  0.006 

No  Alternate  vs  Alternate 

0.26 

1 

0.26 

0.10 

p  <  0.755 

comparison 

Intent  Accuracy 


Complex  No  Alternate  Alternate 

Phrase  Type 

Figure  4.  Intent  Accuracy  Comparisons  for  Phrase  Type 
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4.  DISCUSSION 


The  test  of  the  ITT-1290  speech  recognition  system  resulted  in  the  collection  of  data  for 
8,600  phrase  utterances  and  over  48,000  word  utterances  by  the  1 1  subjects.  When 
summarizing  the  data  for  all  phrases  regardless  of  phrase  complexity  and  number  of 
alternates,  overall  phrase  accuracy  was  slightly  over  95%.  That  is,  95%  of  the 
phrases  had  every  single  word  in  the  phrase  recognized  correctly.  When  phrase  intent 
was  taken  into  account,  intent  accuracy  climbed  to  over  97%.  This  means  that  the 
recognized  phrase  would  have  resulted  in  the  correct  action  being  taken  in  the  cockpit , 
not  95%  of  the  time,  but  actually  97%  of  the  time.  Individual  word  accuracy,  the 
statistic  with  which  most  speech  recognition  researchers  are  familiar,  was  over  99%. 

In  any  speech  recognition  test  such  as  this,  the  accuracy  of  the  system  must  be  viewed 
in  light  of  the  specific  word  vocabulary  and  phrase  structure  being  implemented.  The 
majority  of  work  required  to  create  a  workable  vocabulary  involves  selecting  words  and 
phrases  that  are  not  easily  confused  by  the  speech  recognition  system,  while  at  the 
same  time,  allowing  the  user  (pilot)  ample  flexibility  to  use  a  natural  language  with 
multiple  phrases  that  accomplish  the  same  action.  Allowing  flexibility  in  a  complex  digit 
entry  task,  on  one  hand,  is  very  difficult  to  do.  In  this  study,  there  were  no  synonyms 
for  the  digit  words  zero  through  nine.  This  was  the  reason  that  phrase  accuracy  and 
intent  accuracy  were  identical  for  all  complex  phrases.  The  same  held  true  for  the 
Simple  phrases  that  had  no  alternates  (Simple  No-Alternate  phrases).  The  Simple 
Alternate  phrases  on  the  other  hand  are,  by  definition,  loaded  with  flexibility. 

Alternate  phrases  were  created  to  provide  the  pilot  with  multiple  speech  options  in 
accomplishing  the  task.  The  pilot  could,  for  example,  request  a  cockpit  format  by  using 
a  phrase  that  started  with  either  "show"  or  "show-me."  Different  phrases  utilizing 
optional  words  like  "the,"  "thee,"  and  "for"  could  also  be  spoken.  For  Alternate  phrases, 
phrase  accuracy  and  intent  accuracy  took  on  a  special  meaning.  The  phrase  accuracy 
for  the  set  of  Alternate  phrases  was  only  93.66%.  The  word  pairs  "show"  and  "show- 
me"  and  "the"  and  "thee"  were  the  cause  of  most  of  the  substitution  errors  that 
contributed  to  the  relatively  low  accuracy  statistic.  Other  small  vocabulary  words  like 
"my"  and  "for"  accounted  for  the  most  number  of  rejection  errors.  When  intent  accuracy 
was  computed  for  these  phrases  however,  intent  accuracy  was  found  to  be  99.01%,  an 
increase  of  over  5%.  If  the  recognition  system  failed  to  hear  the  word  "my"  in  the 
phrase  "show  my  hydraulics,"  the  recognized  phrase  still  would  have  resulted  in  the 
aircraft  displaying  a  hydraulic  format  in  the  cockpit.  Intent  accuracy  only  suffered  when 
the  substitution  involved  a  word  that  was  important  to  the  phrase's  intent. 

Given  the  size  and  complexity  of  the  vocabulary  structure  used  in  this  test,  the  ITT- 
1290  recognition  system  performed  admirably,  and  should  continue  to  perform  well  in 
most  PVI  related  simulation  efforts  involving  speech  recognition  tasks.  The  only 
negative  aspect  noted  by  the  subjects  in  the  use  of  this  system  was  the  amount  of 
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template  training  required  to  attain  good  performance.  Three  subjects  also  expressed 
minor  concern  when  the  "TGS"  training  software  quit  operating  in  the  middle  of  the 
training  session,  threatening  the  loss  of  templates  generated  up  to  that  time.  The  issue 
of  template  training  time  and  the  minor  problems  with  the  training  software  should  be 
addressed  in  future  work  with  this  system. 
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Appendix  A.  Vocabulary  Word  List 


Word  Number 

Word  Text 

Word  Number 

Word  Text 

1. 

air-to-air 

26. 

north 

2. 

air-to-ground 

27. 

one 

3. 

AMRAAM 

28. 

point 

4. 

AWACS 

29. 

radar 

5. 

bandit 

30. 

select 

6. 

change 

31. 

setup 

7. 

checklist 

32. 

seven 

8. 

configure 

33. 

show 

9. 

contact 

34. 

show-me 

10. 

cruise 

35. 

six 

11. 

east 

36. 

south 

12. 

engines 

38. 

takeoff 

14. 

five 

39. 

tank 

15. 

fix 

40. 

target 

16. 

flight-lead 

41. 

the 

17. 

for 

42. 

thee 

18. 

four 

43. 

three 

19. 

fuel 

44. 

tower 

20. 

hydraulics 

45. 

two 

21. 

landing 

46. 

waypoint 

22. 

map 

47. 

west 

23. 

maverick 

48. 

wingman 

24. 

my 

49. 

zero 

25. 

nine 
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Optional  Phrase 


Appendix  C.  List  of  392  Test  Phrases 


north  nine  zero 
south  nine  zero 
east  one  eight  zero 
west  one  eight  zero 

north  one  zero  three  one  point  seven  one 
north  zero  five  five  eight  point  zero  three 
north  six  eight  zero  nine  point  four  nine 
north  eight  seven  three  seven  point  six  seven 
north  three  one  four  zero  point  five  eight 
north  eight  two  three  nine  point  two  five 
north  six  eight  four  seven  point  two  six 
north  four  five  two  zero  point  four  six 
north  one  one  five  five  point  six  seven 
north  four  nine  zero  eight  point  four  zero 
north  five  zero  one  six  point  five  five 
north  three  three  three  seven  point  five  four 
north  six  five  four  four  point  five  two 
north  seven  four  five  seven  point  nine  zero 
north  zero  zero  five  eight  point  two  two 
north  eight  seven  one  one  point  one  four 
north  one  four  five  two  point  two  two 
north  one  seven  two  two  point  six  three 
north  three  four  zero  six  point  zero  one 
north  two  six  three  eight  point  three  zero 
north  two  five  two  eight  point  five  zero 
north  three  four  four  zero  point  five  zero 
north  five  eight  four  six  point  zero  six 
north  seven  five  four  five  point  zero  one 
north  five  two  four  nine  point  seven  three 
north  two  eight  five  three  point  one  six 
north  seven  two  zero  two  point  zero  zero 
north  seven  two  three  seven  point  five  eight 
north  four  five  five  four  point  five  six 
north  three  zero  zero  six  point  four  three 
north  two  three  two  one  point  zero  eight 
north  five  two  two  one  point  nine  seven 
north  one  seven  two  zero  point  eight  one 
north  zero  three  four  four  point  nine  six 
north  six  four  one  six  point  four  five 
north  zero  seven  zero  eight  point  six  one 
north  eight  three  one  one  point  one  six 
north  three  seven  two  six  point  four  eight 
north  two  nine  two  two  point  nine  five 
north  four  five  five  four  point  zero  two 
north  five  five  three  nine  point  zero  one 
north  three  five  two  eight  point  three  two 
north  eight  five  two  five  point  five  nine 


north  four  seven  four  nine  point  four  seven 
north  three  seven  one  four  point  six  four 
north  zero  three  zero  six  point  two  six 
north  three  seven  zero  six  point  nine  eight 
north  one  seven  three  zero  point  zero  three 
south  one  two  four  zero  point  seven  five 
south  five  seven  two  zero  point  six  zero 
south  two  zero  five  four  point  six  one 
south  three  six  two  four  point  two  zero 
south  four  two  five  seven  point  eight  four 
south  eight  four  five  three  point  two  six 
south  four  four  five  four  point  nine  seven 
south  one  six  five  two  point  eight  three 
south  five  zero  zero  one  point  zero  one 
south  six  two  two  eight  point  zero  three 
south  two  eight  two  nine  point  zero  eight 
south  five  nine  five  eight  point  nine  eight 
south  four  two  three  six  point  six  five 
south  two  six  three  eight  point  three  one 
south  seven  four  zero  one  point  two  seven 
south  two  four  three  eight  point  six  one 
south  eight  three  five  eight  point  zero  seven 
south  six  eight  one  one  point  four  eight 
south  six  three  five  zero  point  nine  eight 
south  eight  six  zero  seven  point  three  two 
south  one  zero  zero  nine  point  five  eight 
south  eight  four  five  four  point  six  nine 
south  seven  eight  three  one  point  three  eight 
south  six  four  one  eight  point  eight  six 
south  zero  nine  two  nine  point  seven  four 
south  seven  five  zero  four  point  two  eight 
south  five  one  five  four  point  two  two 
south  six  eight  two  five  point  four  four 
south  one  zero  one  three  point  zero  four 
south  six  one  zero  two  point  six  zero 
south  one  four  zero  six  point  four  nine 
south  six  nine  two  three  point  eight  four 
south  five  four  two  two  point  zero  nine 
south  six  two  five  nine  point  three  seven 
south  six  two  zero  five  point  two  seven 
south  eight  four  two  zero  point  one  one 
south  four  six  zero  nine  point  seven  five 
south  seven  one  three  two  point  five  one 
south  three  six  two  six  point  five  five 
south  five  seven  two  five  point  zero  five 
south  eight  four  four  one  point  five  three 
south  six  four  two  zero  point  three  six 
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south  six  seven  zero  seven  point  nine  eight 
south  six  two  zero  one  point  three  two 
south  six  three  five  six  point  seven  zero 
south  two  two  two  six  point  four  four 
south  one  one  four  eight  point  three  seven 
south  two  three  five  six  point  four  nine 
east  zero  two  one  three  four  point  six  nine 
east  one  five  five  one  zero  point  four  one 
east  one  one  nine  two  six  point  five  one 
east  one  four  three  five  five  point  eight  five 
east  zero  nine  zero  four  one  point  one  nine 
east  zero  three  nine  one  zero  point  four  nine 
east  zero  five  one  four  six  point  three  two 
east  zero  five  five  three  two  point  nine  three 
east  one  two  zero  four  three  point  two  three 
east  zero  six  seven  one  one  point  zero  zero 
east  one  six  six  four  six  point  eight  eight 
east  zero  eight  six  one  six  point  five  five 
east  one  one  nine  two  one  point  three  four 
east  zero  one  eight  five  three  point  five  seven 
east  one  six  five  three  four  point  seven  six 
east  one  six  eight  three  six  point  eight  one 
east  one  five  one  four  six  point  one  four 
east  one  two  zero  one  seven  point  nine  one 
east  zero  seven  zero  zero  two  point  eight  nine 
east  zero  five  three  two  eight  point  six  five 
east  zero  zero  six  four  three  point  six  six 
east  zero  six  nine  zero  three  point  eight  two 
east  zero  nine  two  five  zero  point  two  nine 
east  zero  one  six  one  seven  point  zero  two 
east  zero  five  seven  three  five  point  eight  three 
east  zero  seven  two  four  nine  point  nine  two 
east  zero  three  eight  zero  one  point  one  six 
east  one  two  five  five  three  point  one  two 
east  zero  six  seven  five  two  point  three  one 
east  one  zero  zero  two  seven  point  one  five 
east  one  five  nine  zero  four  point  six  zero 
east  one  zero  nine  three  two  point  four  nine 
east  zero  seven  four  two  three  point  one  six 
east  one  zero  six  three  nine  point  zero  three 
east  one  six  eight  zero  three  point  zero  seven 
east  zero  nine  two  five  zero  point  seven  four 
east  zero  zero  one  three  three  point  seven  two 
east  one  seven  seven  five  one  point  six  eight 
east  zero  eight  nine  three  two  point  four  three 
east  zero  nine  seven  one  zero  point  eight  eight 
east  zero  four  two  five  eight  point  seven  one 
east  one  two  zero  three  zero  point  two  three 
east  one  five  two  zero  one  point  one  one 


east  zero  eight  three  one  five  point  three  eight 
east  one  four  two  zero  nine  point  six  six 
east  one  four  zero  three  five  point  one  three 
east  zero  zero  seven  three  three  point  six  nine 
east  zero  zero  one  zero  three  point  eight  four 
west  one  two  six  five  seven  point  three  seven 
west  zero  five  four  zero  three  point  three  seven 
west  one  zero  seven  two  nine  point  nine  three 
west  one  one  one  one  two  point  three  one 
west  zero  nine  five  zero  nine  point  five  nine 
west  one  six  four  three  one  point  seven  five 
west  zero  three  seven  two  nine  point  nine  four 
west  one  zero  five  zero  seven  point  three  six 
west  zero  five  two  five  two  point  three  eight 
west  zero  four  five  three  three  point  seven  seven 
west  one  zero  five  two  zero  point  eight  nine 
west  zero  four  eight  two  six  point  two  three 
west  zero  five  three  zero  six  point  four  three 
west  zero  seven  one  two  one  point  nine  eight 
west  zero  five  three  four  two  point  six  six 
west  one  four  six  zero  six  point  four  zero 
west  one  one  four  four  eight  point  nine  one 
west  zero  eight  zero  one  zero  point  five  eight 
west  one  zero  three  three  four  point  six  four 
west  one  three  six  zero  nine  point  zero  five 
west  one  four  six  four  two  point  seven  three 
west  one  five  four  three  five  point  three  zero 
west  one  one  five  two  nine  point  six  six 
west  zero  eight  eight  three  eight  point  zero  nine 
west  zero  six  five  three  seven  point  nine  five 
west  zero  four  one  three  six  point  one  zero 
west  one  six  seven  two  six  point  two  six 
west  one  six  zero  one  six  point  four  two 
west  one  four  five  one  zero  point  nine  one 
west  one  four  nine  one  four  point  eight  one 
west  zero  eight  eight  zero  one  point  one  five 
west  zero  zero  one  two  five  point  three  two 
west  zero  three  three  five  four  point  one  two 
west  zero  four  eight  two  four  point  three  nine 
west  zero  three  three  three  nine  point  nine  eight 
west  zero  zero  zero  one  four  point  six  three 
west  zero  one  nine  five  zero  point  three  two 
west  zero  seven  nine  three  four  point  three  eight 
west  one  four  six  four  zero  point  eight  six 
west  one  zero  two  three  five  point  five  seven 
west  zero  seven  six  four  four  point  one  two 
west  one  six  seven  zero  four  point  nine  seven 
west  zero  six  six  two  two  point  eight  one 
west  zero  zero  seven  two  five  point  eight  nine 
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west  one  four  six  two  seven  point  four  one 
west  zero  four  two  two  six  point  nine  six 
west  one  five  one  five  eight  point  three  three 
west  one  one  five  one  seven  point  eight  three 
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change  waypoint  one  select  waypoint  nine 
change  waypoint  two  show  bandit  one 
change  waypoint  three  show  bandit  two 
change  waypoint  four  show  bandit  three 
change  waypoint  five  show  bandit  four 
change  waypoint  six  show  bandit  five 
change  waypoint  seven  show  bandit  six 
change  waypoint  eight  show  bandit  seven 
change  waypoint  nine  show  bandit  eight 
select  amraam  show  bandit  nine 

select  maverick  show  radar  one 

select  air-to-air  show  radar  two 

select  air-to-ground  show  radar  three 

select  bandit  one  show  radar  four 

select  bandit  two  show  radar  five 

select  bandit  three  show  radar  six 

select  bandit  four  show  radar  seven 
select  bandit  five  show  radar  eight 

select  bandit  six  show  radar  nine 

select  bandit  seven  show  tank  one 
select  bandit  eight  show  tank  two 
select  bandit  nine  show  tank  three 
select  radar  one  show  tank  four 

select  radar  two  show  tank  five 

select  radar  three  show  tank  six 
select  radar  four  show  tank  seven 

select  radar  five  show  tank  eight 

select  radar  six  show  tank  nine 

select  radar  seven  show  waypoint  one 
select  radar  eight  show  waypoint  two 
select  radar  nine  show  waypoint  three 
select  tank  one  show  waypoint  four 

select  tank  two  show  waypoint  five 

select  tank  three  show  waypoint  six 

select  tank  four  show  waypoint  seven 

select  tank  five  show  waypoint  eight 

select  tank  six  show  waypoint  nine 

select  tank  seven  show-me  bandit  one 

select  tank  eight  show-me  bandit  two 

select  tank  nine  show-me  bandit  three 

select  waypoint  one  show-me  bandit  four 

select  waypoint  two  show-me  bandit  five 

select  waypoint  three  show-me  bandit  six 

select  waypoint  four  show-me  bandit  seven 
select  waypoint  five  show-me  bandit  eight 
select  waypoint  six  show-me  bandit  nine 
select  waypoint  seven  show-me  radar  one 
select  waypoint  eight  show-me  radar  two 


show-me  radar  three  show-me  my  checklist 
show-me  radar  four  show  fuel 

show-me  radar  five  show  hydraulics 

show-me  radar  six  show  engines 

show-me  radar  seven  show  the  fuel 

show-me  radar  eight  show  the  hydraulics 
show-me  radar  nine  show  thee  engines 
show-me  tank  one  show  my  fuel 

show-me  tank  two  show  my  hydraulics 

show-me  tank  three  show  my  engines 

show-me  tank  four  show-me  fuel 

show-me  tank  five  show-me  hydraulics 

show-me  tank  six  show-me  engines 

show-me  tank  seven  show-me  the  fuel 

show-me  tank  eight  show-me  the  hydraulics 
show-me  tank  nine  show-me  thee  engines 
show-me  waypoint  one  show-me  my  fuel 
show-me  waypoint  two  show-me  my  hydraulics 
show-me  waypoint  threeshow-me  my  engines 
show-me  waypoint  four  fix  fuel 
show-me  waypoint  five  fix  hydraulics 
show-me  waypoint  six  fix  engines 
show-me  waypoint  sevefix  the  fuel 
show-me  waypoint  eightfix  the  hydraulics 
show-me  waypoint  nine  fix  thee  engines 
show  stores  fix  my  fuel 

show  map  fix  my  hydraulics 

show  target  fix  my  engines 

show  checklist  configure  takeoff 

show  the  stores  configure  landing 

show  the  map  configure  cruise 

show  the  target  configure  for  takeoff 

show  the  checklist  configure  for  landing 

show  my  stores  configure  for  cruise 

show  my  map  setup  takeoff 

show  my  target  setup  landing 

show  my  checklist  setup  cruise 

show-me  stores  setup  for  takeoff 

show-me  map  setup  for  landing 

show-me  target  setup  for  cruise 

show-me  checklist  contact  tower 

show-me  the  stores  contact  awacs 

show-me  the  map  contact  flight-lead 

show-me  the  target  contact  wingman 

show-me  the  checklist  contact  the  tower 

show-me  my  stores  contact  thee  awacs 

show-me  my  map  contact  the  flight-lead 

show-me  my  target  contact  the  wingman 
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contact  my  tower 
contact  my  awacs 
contact  my  flight-lead 
contact  my  wingman 


Appendix  D.  Subject  Briefing 


The  purpose  of  this  study  is  to  test  the  recognition  accuracy  of  a  new  connected  speech 
recognition  system. 

During  the  next  hour  or  so,  you  will  be  asked  to  use  the  microphone  on  the  headset  to 
speak  the  phrases  that  are  presented  on  computer  screen.  There  are  329  different 
phrases  that  will  be  presented.  You  will  always  be  prompted  when  to  say  the  next 
phrase. 

The  first  20  minutes  of  the  study  involves  "training"  the  voice  recognition  system  to 
recognize  your  voice.  We  do  this  by  prompting  you  to  say  individual  words  and  then 
phrases  that  contain  these  words.  When  you  speak  each  word  or  phrase,  the 
recognition  system  will  remember  how  you  spoke  so  that  it  can  recognize  the  words 
and  phrases  the  next  time  it  hears  them. 

The  second  part  of  the  study  involves  letting  the  voice  recognition  system  actually 
recognize  your  voice.  Again,  the  phrases  to  be  spoken  will  be  presented  on  the 
computer  screen,  in  random  order.  We  will  take  a  short  break  after  approximately  130 
phrases,  and  again  after  260  phrases. 

Due  to  the  amount  of  reading  required  during  this  session,  you  may  on  occasion 
misread  a  phrase.  If  this  happens,  I  will  simply  present  that  phrase  again. 

You  are  a  volunteer  subject.  Participation  in  this  study  is  your  choice.  If  at  any  time 
you  wish  to  discontinue  the  testing  and  your  participation,  you  may  do  so.  In  case  of 
fire  or  emergency,  leave  through  the  CSIL  main  entrance. 

Are  there  any  questions? 
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Appendix  E.  Summary  Data 
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Grand  Total  for  Trials  la;^  ||  4,398.00  |  4,406.00  |  99.82%  ||  773.00  |  784.00  |  98.60%  ||  777.00  |  784.00  99.11%l  10000%  |  10000%  |  9821% 
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Summar 
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